Skip to content

Add bench mode #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
chenyushuo merged 5 commits into from
May 12, 2025
Merged

Add bench mode #35

chenyushuo merged 5 commits into from
May 12, 2025

Conversation

@hiyuchang
Copy link
Collaborator

@hiyuchang hiyuchang commented May 12, 2025 •
edited
Loading

Description

  1. Add eval mode. For now, the test data is passed through eval_split.
  2. Add a unified logic for trainer_buffer reader -> Stop if no task is available.
  3. Fix a bug in DPO batch_size computing

Checklist

Please check the following items before code is ready to be reviewed.

  • Code has passed all tests
  • Docstrings have been added/updated in Google Style
  • Documentation has been updated
  • Code is ready for review

@hiyuchang hiyuchang changed the title Add test mode Add eval mode May 12, 2025
@pan-x-c
Copy link
Collaborator

pan-x-c commented May 12, 2025

/run-unittest

@hiyuchang hiyuchang changed the title Add eval mode Add bench mode May 12, 2025
@github-actions
Copy link

github-actions bot commented May 12, 2025

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
22 22 0 0 0 0 0 830ms

Failed Tests

No failed tests ✨

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name Status Flaky Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer ✅ 3ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer ✅ 1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid ✅ 1ms
tests/common/config_test.py::TestConfig::test_load_default_config ✅ 1ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion ✅ 1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion ✅ 1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate ✅ 36ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate ✅ 41ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate ✅ 55ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate ✅ 56ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate ✅ 39ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask ✅ 1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer ✅ 1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer ✅ 167ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer ✅ 188ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool ✅ 18ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow ✅ 1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow ✅ 1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow ✅ 1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow ✅ 1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer ✅ 1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer ✅ 215ms

Github Test Reporter by CTRF 💚

@chenyushuo chenyushuo merged commit d613940 into modelscope:main May 12, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chenyushuo chenyushuo chenyushuo approved these changes

@pan-x-c pan-x-c pan-x-c approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hiyuchang @pan-x-c @chenyushuo